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ABSTRACT 



/ • The present report is a follow-up on third year pupils served by the 
reading center project during the 1972-73 schopl year. The original, study showed 
pupils who -participated in the project tended to score higher on- reading tests 
than would iipparentTy have otherwise bden the case. The present study sought to 
document whether or not these apparent gains would still hold up about sixteen . 
months'after participants left. the program. Another less rigorous study conducted 
several -^ye&rs ago had indicated such gains had held up'. The present study also 
supportedZthis contention'. There was, however, in this studjr a somewhat consis- 
tent but nonsignificant (in a statistical sense) tendency for gains to be main- 
tained better among black than white pupils,.- " . . ' 

\ ■ The predecessors of this study helped focus national attention upon the 
Vocal' reading center program. The American Institute of Research (AIR) used the 
documentattoti which has been accumulated regarding the effectiveness of this pro- 
gram as a basis for identifying it as an exemplary program. The findings of this 
report basically reinforce the conclusions of the previous studies. 

Additional supporting evidence regarding the' effectiveness of this 
program Was provided by !a recently completed evaluation of the loca.1 ESSA program. 
Methods and materials developed and used at the reading center were applied to 
over 2500 local first and second year, pupils identified as. needing help m reading 
on the basis of low test scores. The evaluative evidence indicated that the ESAA 
pupils so served, performed- better in reading than would have otherwise been the 
casev' Without the reading center, such results might not have been obtained. 

The conclusions of this report*"were designed to promote constructive 
and critical' thinking. Even though the^evidence was strong that the reading 
center was benefiting tfiose served, the criterion for selecting participants 
was'chaTjged during the 1973-74 school yeaf. Beginning then, those most iti need 
rather than those' "ready to benefit" from these services were served by the 
center'. Although the ESSA evaluation seemed to indicate that pupils "most in 
need" could benefit from methods ahd materials developed at the center, no hard 
evidence) is presently avai lab-le to document whether or not the program is equally 
effective with pupils selected on the basis of the new criteria. The type of ■ 
"performance data" presently used to evaluate the program cannot provide such 
"hard" evidence. In any case this programs recognition by AIR should serve to 
alert decision -makers at the local ,- state and federal level as to just how .rare 
and unconrxjn it is for a program such as this to be ab.le to document its success 
with "hard" evidence. Although it is important to find out if the program is , 
equally successful in serviTig pupils; on a ''most in n^ed" basis, the- program s 
effectiveness with pupils meeting its previous criteria for selection points out 
that compensatory fundinc should be allocate<i to insure that all local pupils, 
who can benefit from reading center services, do in fact receive these services. 

Finally there is a need for further reflection by decision-makers re- 
garding the contributions ».which can be'made by those familiar with tests and 
measjuKments and evaluative techniques to substantiating and implementing local 
projiits'. For example, an evaluative technique which is useful for one purpose 
may be quite unacceptable when extended into Another area. Also tests^are not 
infallible and, if used as selective devices, they must be supplemented by other 
selective criteria. Otherwise pupils wiJ.1 be selected, for project participation 
who should not be and others wi.ll be left out who should be included. 



. ' READING CENTER: 1972-73 FOLLOW-UP STUDY 

INTRODUCTION: ? 

This study's purpose was to assess .the long run benefits of partjqilpation 
in the. reading center program. The readihg center program features a cnnfcal 
approach with an emphasis Mpon diagnosis ^nd individualization.* The jeading^^ • 
center has been continuously evaluated by this Jdepartment since the l^eF-es school 
year Results of previous studies were essentially supportive of the center in 
that it appeared participants made significant gains in reading skills which they^ 
still maintained one year after being in the program. This report is organized so 
that the main body is directed to the lay reader. The more deftailed and tedious 
material is appended". . » 

PROCEDURES: j ; . 

^ The 1972-73 school- year presented local evaluators with a somewhat unique 
opportunity to implement a more Vigorous research design than had been possible in 
Previous years. . Because more pupils were eligible for project participation than 
the program could house, it was possible to randomly select a control group from 
the list of all eligible pupils. 

'•- The "magic" of randomization is that it provides an effective control for 
all extraneous factors which could be associated with ^ij^t^?;f P"P^J^^„ 
test scores. The 1972-73 sliudy**.* therefor6 provided a hlgMy defensib.le assessment 
of at least the short term benefits of the.-project. The present study was built 
in" to' the design of the 1972-73 study.' In addition to randomly selecting a con- 
trol group (control group #1) a second control group was selected from seven schools 
whi'ch were not served by Title I. The pupils in the second control group (Control 
group #2), wbre selected in September/1972 on the basis of screening tests- admin- 
istered by reading center personnel/ 

Because entry Into the project of pupijs in control group #1 was -Relayed 
until summer rather than denied/it was thought that control group #2 would serve 
as the comparison group for the presently Reported longitudinal study. 

■ I - The results of the 1972-73 study indicated that a random sample of the 
third year pupils served by the project attained a higher level of reading pro- 
ficiency than would have been ikely had they not entered f ^ Program. This in- 
creased^roficiency was maintained- four months subsequent to pupils ^ismssa from 
the program. The present study was designed to see if participants st 11 ""ajntajned 
their advantage about sixteen months after leaving the program. In brief, had the 
gains "washed out" over time? 

As it happened, only a few members, of control "group #1 took advantage 
if their opportunity to participate in the program during the summer. Thus p^r|i- 
cipantS' could be compared with both control groups.. All testing in the 1972-73^ 
study was carried out by reading center personnel. The .present study was 
strengthened in that the criterion reading tests were administered completely 
iSendlntly of the reading center as part of a countywide testing program. The 
' standardised tests used in the countywide testing were not .the same as those used 
by reading center personnel .. - 

*Persons interested in a detailed description of the program may contact Miss Louise 
.Sears, Reading Center, The School Board of Broward County, Fort Lauderdale, Fl a. . 
**See Research Reports #15^ 29, 35, 36, 51 and 67. . \ . 

***Report #68. ^ ^ ' . • ^ 



LIMITATIONS? / 

"Because of attrition/ the findings could only apply^o pupils still 
enrolled in the county. The reading center-had for several years served the 
schools from which control group #1 was randomly drawn. Becafjse of this, com- 
parisons between control group #1 and participants were- quite likely subject to 
a limitation ref^red to by/researchers as "contamination." Were pupils in con- 
'trol group #1 really the same as they would have been had the reading center not 
existed? ,It is plausible/that these pupils received indirect benefits due to the 
reading center before, djdring and after the •^tiTC of the study. 

On the other/hand, comparisons between (control group #2 and participants 
were' subject to a limitation researchers call' "confounding." Even though statis- 
tical procedures wer6 usAl to "control, -for" pre'-test differences, factors su^h as 
socio-economic differences between the pujbil populations of Title I and non 
Title I schools could have- influenced differences in post-tiest perfor^mance. This . 
"confounding" prevented one from being certain that the only difference between 
the participants and control group #2 was entry into the reading center. 

Such limitations are inherent in almd^t'any practical study of ongoing 
educational,progran)s. '"In the present study, the fact that the two control groupsr-^ 
Were .available' provided an opportunity to sep if findings of bo.^h comparisons' 
were congruent. The study was also strengthened in that a previous study had 

Jicated long ruh as well as short run gains. The congruence of the findings , of 
dies conducted at different times and using different approaches, provides what 
probably the strongest case far drawing justifiable conclusions from the appn- 
■ cation of research methods in educational contexts. In the case of the reading 
center, yet another source of supporting evidence became available at the time 
this' report was written. Methods and materials Used by jTeading, center personnel 
were- bodily incorporated into the local Emergency School Aid. Act (ESfAA) reading 
program implemented during the 1973-74 school year. Results..of evaluations of 
the ESAA program were therefore also pertinent to building a body pf congruent , 
findings re^ar/ling at least the short tem effectiveness of the reading center 
^ program. ;These ^finding? wi,ll be discussed fn the conclusions section. ' ■ 

RESULTS , ^ ^ ■ '' • 

Comparisons involving the first control group were clear-cut in the area 
of reading vocabulary. The experimental group scored significantly higher than V 
control group #1 on the reading vocabulary test. Although findings on the reading 
comprehension test tended to be favorable to center participants, results were not 
statistically significant. " Further, one could interpret the favorable trends in 
eitner of two ways. ..One interpretation would be that differences between white 
pupils tended ta "wash out", but black pupils who participated tended to maintain 
their "edge" over the control group. This tendency approached statistical signi- - 
ficance. The t^t.of^simple differences, collapsing across races, also approached 
statistical .significance. , 

* . ' 

Comparisons involving the second control group on the reading vocabulary" 
test revealed the presence of a highly complex relationship involving race, treat.- 
^ment and pre-test scores. Hdwever, apparently pupils of both races who scored low 
on the pre -tests .and niost black pupils benefited from participation in the program. 
On the reading comprehension test, simple overall differences favoring participants 
wel'e statistically significant. However, again there was a tendency for program 
participation to benefit blacks more than whites which approached, but did not 
reach, statistical significance. 

■•>>,• , • ' • 



t CONCLUSIONS • . . 

The 'findings of this study seemed to indicate^that participants' gains 
tended to be maintained sixteen ijionths after finishing the program. There was \, 
some, but by no means conclusive, evidence that black participants may have 
benefited more in the Jong run than did white participants. 

' ^ ■ Further congruent evidence regarliing at least the short run impact of 

the reading center upon couirty pupils was provided by another study completed inv 
- / the fall of. 1974.* -'Methods and itiat6|rials used in . the clinical-remedial program 
at the reading center were used in a preventive-developmental program which served 
over twenty-five hundred 'first and second year pupils from thirty-local schools 
whoVhad initially scored very low on standardized reading readiness tests. It was 
found that pupils who were so served appeared to perform better on criterion 
reading tests than would have otherwise been the' case". Thus, yet another source 
of data pointed in the same direction as the present study and previous studies; 

Beginning Wilth the 1973-74 school year the reading center altered its 
screening pracedures. 'I'rior to that time the center essentially served pupils* 
with low achievement Who had the pWntial to do better. Perhaps it would be 
accurate to say that '-^readjness" ^as the criterion for program entry. .In 1973-74 
the program began .to serve 'pupils^ on the basis of "most in need." Pupils with 
the lowest scores on standardized tests were the ones eligible to enter the pro- 
gram with little regard for other factors. It might be mentioned that the ESAA _ - 
•project and the reading center both served pupils on the basis of "most in need.. 
, Both therefore had to develop "readiness" components for pupils who were not 
sufficiently advanced to be exposed to the methods and materials which have pri- 
marily constituted the p-rogram component which the research department has eval- 
uated* over the years-. The results of the ESAA project evaluation were, encouraging. 
However, it has'not yet been possible for this department to document the impact 
of the reading center program upon its current enrollees, • , 

• . ' * 

The research evidence has provided a strong case for concluding that^ y 
' pupils benefited from the reading center program prior to the change in selection ' 
procedures. Furthfermore, the ESAA evaluation provided a strong hint J:hat methods 
N and materials developed and used^at the reading center could -be effectively 

"exported" to in-school pr/ograms serving primary level pupils who^ were identified 
as being at the lowest levels in terms of measurable reading skills. 

Plans are currently underway\o implement the types of in-depth research- 
studies of the current readtng center project as have been done in the past. What- 
ever the Yesults of future studies, however, the weight of the evidence suggests 
\5trongly that a significant segment of local, pupils have benefited from the^reading 
center program. The recent national attention focused upon the reading center as 
a result of the American' Institute of Research (AI/)'s efforts tojdentify 
exemplary reading programs should suffice to indicate how uncommon it is for such 
a program to be able to rigorously document its effectiveness in terms of pupil 
outcomes. This report further reinforces the documentation. Therefore, even if 
sifbsequent evaluations indicate "the program is not as effective when implemented . 
on a serving those "most in need" basis, the results of the research to date should 
^ be put to positive use. I . 

There^may well be aVmeed for local, state, and federal officials as well 
as advisory groups, to pause md reflect upon questions such as "Are some pupils 
who could be helped by this program now bein^g excluded by the new criteria?" 

★See 1973-74 Final Technical Progress Report. Emergency School A'id Act (ESAA) ,. 
Basic Grant - OEG-4-73-1493. f, '■ ^ 

ERIC 



"Are group administered standardized test results. really so Infallible that^heyy 
cdn serve as a good basis fgr Identifying" pupils most in need of help?" "Is there 
anything to the concept, of maturational readiness and,, if so, what, where, and 
how should programs be developed and implemented for. pupils who are slow to ^^^^ 
develop?" ■ ' 

Another area^^wtnch also deserved careful "attention is the currertt 
emphasis upon attaini/q po^^wnce pbjectives as an evaluative device. Performance 
objectives provide a xer^fesflficial means of- ,yalidatinq what a program can be 
'expected to achieve. For&COm^, test results might show that pupils win gain 
"'a certain number of qrade equTv^^ent months after profram participation »x-Others 
considering implementing such a program with similar pupils can ^expect to attain'" 
similar results. Given accurate informa.tion based' upon research or previous 
experience ^ith the same population and instrument, it also makes sense to set 
realistic objectives and make their achievement an importan;5:-^luative criteron. 
Simple analogies with toothpaste ads may clarify the logi^^of the above. Use of I 
brand A by a certain population may be associated with an^average of X number of ' 
cavaties per, person over a specified time. This validates for other members of 
that populatipn what they might. expect from using brand A. Of course, to use this 
data to evaluate brand A would require knowing the normal rate of cavaties per 
person in that population over the same time spap. The danger of using performance 
objectives lies ifi making unwatS:;anted evaluative assumptions, often at the stage J 
of specifying objectives. Persons who are not at home in the area of tests and. ^ 
measurements cannot be expected to be familiar with the many traps and pitfalls 
associated v/ith tryi-ng.to accurately measure changes in test performance. 
Because of this, the extension of the cl^'ms Of essentiaUy validitatio nal studies 
into the evaluative domain can often be ^demon^tratjESd to produce erroneous' con- 
clusions. For example, the gain of a no>m group tannot be compared with the gain 
obtained by a certain subpopulation selected on thr basis of receiving low initial 
test results. Unless a' test is perfectly reliable, negative ejTfors of measurement 
wiJ^ occur a disproportionate number of times in-the s.elected/group. The gr~oup 
as a whole can be > expected with mathematical certainly to score higher on a second 
administration of the test. In' brijef , some of 'the. "gain" between pre and post 
tests obtained by a group so sele^^ed is due to errors of measurement rather than 
program benefits. Furthermore, if projects use scoring low on one fallible 
standardized test as -the single criterion for selecting pupilSo"most in need ; 
many children will be selected wh^,.^ually are not as much in nefed as many 
children who a re not selected.^^/"'^ • • 

The above statements only reflect on the nature of tests' and measurements. 
However, they do lend support to the need for systematic program evaluation to 
assist in making major educational decisions ,whieh will influence the welfare of 
pupils in need of compensatoiy services. For example, a strong case can be made 
in terms of the present evidence for assuring that pupils demonstrating low achieve- 
ment with a potential to do better receive -the ktnds of services offered by the 
reading center, as our evaluative e\ddence' shows that these students do benefit / 
significantly from the services. An^qually stronq case can be made for demandi/hg. 
that siiyllar hard evidence be" secured whenever possible to substantiate all simil 
projeclfs at the local, state and national leveW A correlary of this is that a 
hard look needs to be taken at various means of evaluating such prpjects in order 
that decision makers can be informed and confident abbut how much credence to- p1!i 
in evaluative claims. 



*3ee for example, Chester Harris, ed. Problems in Measuring Change, The University 
of Wisconsin Press, Madison, 1967. 
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APPENDIX 



PROCEDURES: v / . ■ » , • . ^ . ^ 

Two control group were se.lec/ed in the" 1972-73 study. One fiorvt'rol (group •. 
(control group #1) was randomly selecjfed from -the- pooj of third year pupfl s eH igible 
for entrance to 'the program. AvailaMe facilities could not .accommodate all; eligible- 
pupils. It was therefore, made stricily a matter of chano^. as to which /pupils would . 
not receive program services dii,ring/the regular school .ye^r. The progridm was opfen , ^ 
to control group #1 pupIVs thit' sun/ner. Hwyever, a ^heck gf records revealed that 
only five pupils in control group /l entered the reading cen^ter thaysummer or during 
the subsequent year. Control' gro/p #2 was/selected by reading center personnel who • 
employed regular prjjject screenir|| tests /during. September,;^! 972., ip seven, nonproject 
schools. Third year pupils in t/e. nWoject schools who ^et thS/Scr^emng Cf-iteria • 
for project participation compo/ed control group #2.^e>xperim6ntal. group con- 
sisted of a random sample of 'third year pupi ls 'enrol led.-aV the r.ea.dmg center during 
the fall of the 1972-7S school /year. , ?/ 



\ri 

i/ng 



marv Reading Tesf and~ t|^6 mtis s)iort form IQ test' 
I . cr i ter i a . used in September/. : Jhe September, 1972, 



. Form A of .the Gates, 
constituted the initial. sereeJ.y.-j. w. .w^. ^ 

reading scores on the Gates. tMt jyere used as^re-tests/in tfhe present study. P(Rt, 
test reading scores were obtained from a cpujit>-wide testing program carried out « 
May, 1974. The Comorehensi^ Tests of Basic Skills (CTBS ) / expanded edi tion, 
level* 2 was the test used- 1|fi 'the count-wide administration. ; . 

' ^Student I.D. numliers were use'd to match counjtywide test/r-esWts- with data-v^./ 
records used in the preceding reading Center study, -liable I illustrates the^;^ - ^ . 
•attrftion whiijh occurred dLring the course of the evaluative period, ^^ye of the . 
Students' tn control group]#l Were elinfinated because they later entered the program. 

TABL-E- 1 . X' ' 

ATTRITION. 



Oriqirial Sample ^ First Study (fw) 



Control Group #1 
Control Group #2 
Exp. Group 



.LIMITATIONS: 



'60 
128 

119" 



58 
116 
107 



Fol Tow-up Study (May) 



41 

95v 

82 



the experimental group were ' 



Comparisons between control group #1 and . _ , - . ■ . . 

soundest from a methodological viewpoint. Membership in 'these groups depended upon 
random assignment from a common pepulal^i on. The principal li^mitation upon such,. ^ 
comparisons centers upon the problem of contamination. The reading center had 
served project schools folb many years. . It was therefore' questionable- as to whether 
^or not the children in cd^trol groupffl^ad not already -received indirect benefits . 
from the reading center. ^They may well have not sdpred aS high, for example, on 
theiV pre tests had the reading cSnter prpgraifi not existed in previous years. . 

\ jri^ principal limitation of comparisons involving control Group #2 was 
i>ou^!iifg.' Differences betv/een, this group and -the experimental ^roup could have 



been du#to factors other than participation in the reading- center program. For 
•exampl^ the population factors which determined which spools would be designated 
as Title I schools could obviously be. an- independent soilrce of- differences in test 



results between control group #2 and the extjerimental group. A.Uhough .the same - 
statistical procedures were used in making the comparisons, the interpretation 6f 
the analyses differs. In the case of comparisons invofving control group #2, it 
was posited that pre-test scores were, probably somewhat associated with other con-' 
foundirtg factors. The statistical ^tests simply. attempted ft isoVate >:he conttri-- 
biiticn of the program which was not confounded with race and the pre-test iscores. ^ 
In the comparisons with* the first control group, pre-test differences and differences 
in the proportion of black pupils jn each group could be recjarded as^ arising due tp . 
sampling errorsT rather than confounding/ factors. • t . C* ^ 

As has been noted, both procedures had inherent limitations. Concurrence 
of results from both compariscfns v^ould, of course, provide the most acceptable?/ 
evidence regarding the pr|^ence or absence of a significant program e/fect. y 

The problem of attrition simply limited the generalizations of t^h6 find"- 
1ngs to pupils like the ones who could be located^or .re-tefsting. - \ , 

, ' One limitation of the preceding study was el'iminhted in the present ^' 

foliow-up study. The criterion test in the present study^was not administered by 
.reading -center personnel . , , i 

Finally, it must be noted that the reading cen^rVogram ha? be^n con- V 
tinuously evaluated since' the 1967-68 school year.* A pr^^oas, less rigorous,^ 7* 
follow-up study had indicated that the program had seemed to produce long term- 
benefits to pupils. So long as the results of this study rsmained consistent with 
the previous one, the cummalative evidence would appear to jlj^ify drawing fairly 
strong conclusions, from the present study, despite the aforeinek|:iori^ limitations. 

RESULTS 

DESCRIPTIVE: 

Mean's (averages) and standard deviations for the three grob^^ps involved 
in' this study^are summarized in Table II. Groups are further. subdivided by race. 

• TABLE II ■ ' 
Means and Standard Deviations (Raw Scores) 



r^est|_ 

^ lead Cc 

tace^' Treatment N "TT" M_ "ST" M ' W' M ' SD 



Sates Pre-Tests . CTBS Post fest^ 

Groups ^ Read Voc Read Comp Read Voc Read _ Comp 



White Exp 34 23.3 6.5 12.1 4.5 22.7 7.6 "23.9 ' 10.0 , 

WhiJt Corit iS^l 19 27.1 7.4- 13.7 4.7 23.5 ^,8.4 26.1 10.6 
White Cont#2 54 22.1 '^7.4 11.5-5.0 22.1/ 8.5 22.7 9.6 
Black Exp • 48 21.0 6.3 12jr9 4.8 22.T 6.9 24.1, 8.8 

Black. ' Con t #1 . 22 24.5- 6.5 12.8 3-.3 19.9 7.2 19.5 9.1 
Black Con t 15*2 41 }6i8 7.5 10.0 4.9 ^ T6.8 8.4 16.2 8.6 
' '■ ■ ■ ■ ' 

Table 11 indicates that the. surviving members of control group #1 tended 
■ to score higher on the pre-tests tharTdid other groups. It would go Deyond the 
avail ^le facts to attempt to explain trfis phenomena. However, some. of it was ' 
due to sampling error, because the same tendency was present in the preceding 
study independently of the attrition factor.** It was also the case m the 

" *See Research Department Reports Nos. .15, 29, 35, 36,^5^, 67. 
^ ' -,**See Report #60, page 3. ^ i) 

ERIC ^ . * 



■'"preceding' study that controVnroup #2 'fell below the otheygroups on th* P';^:^"*^- 
pirt of the reason for these ►^re- test differences fi^v.orinj gVoups from Project 
.schools may.have 'b^cn the "cbntaMnatinq^ effect of the reading . '^^^^^^.^^ 

.arouD inav h*ve done so weU fiad their schools not been served by the reading center 
X the ye??1. IThe particularly l,ow. performance of black p'upils n control group 
-#2. Wbfe of some substantive significance. V^re fewer black pupils in P^oJect^ 
schoolJ^'whdyre identified as needing help in Veading. performing as low as similar 
pupils l,nn on project schools? ' • . . ' ' • 

' Study ot Table II can help in^ interpreting:, some of the statistical find- 
ings which follow. For ey?flnple, a ten'dency can be discerned for black exper menta 
Jupils to have dBne relatively better on the post tJsts than did white experimental 
'ils, notwithstanding pre-test differences. 



StATISTJC^ ANACjffiES^: 



• "AjveHion of a MANpVA program developed by Eliot Cramer was used the 
analyses wMr<h f'oVTow.^ A more detajled summary of t,he statistical tat^ es is 
appended, factor analysis of ^ovariance design was used which^ol lowed the 

basic logic^of itodel reduction inherent in the MANOVA program.- The two factors 
were raciahd- treatment. Pre-tfest reading vocabulary and comprehension sdores 
were covaried. The MANOVA logic for model reduction is one which precedes from 




the complex to the simple. 

Three tests of significance were of, relevance to the present study. 

1. Test of the equality of cell regression coefficients: 
Essentially, this- is attest of a complex ilfteraction r^l nvolving both 
factors and the two covariates. .Significant results would show 
that the post test results depended simultaneously upon how well 

■■ pupilsTdid on the pre-tests as we 1.1 as their race- treatment group 
membership. < , - . " 

2. Test of the race-treatment interaction:' » 
This is a testVor an interaction between the two group membership 

> factor?. For example, blacks in the experimental group might have 
^ fared, relatively better -than wliites in the experimental group'. 

3. ' Test for the) treatment main effect: ^. * 
■ This is a test of the simple difference between all members of 

' * the two treatment group. , - 

' . ■ ' , ■ ^ 

•• ■ The MANOVA logic dictates th?t proceeding with tests of simpler models 
•^epends upon rejecting' the preceding tests 6f more complex models. PJ. Jor ^ 
/example;, the model which be»t fits the factS'(or at least cannot be rejected at 
/the .05 level of^ignificance), involves a complex interaction, the analyses cea 



cease. 



EXPERIMENTAL GROUP VS.. CONTROL GROUP #T 



ERIC 



^dtaalyses proceeded past the test of equality of regression coefficients. 
The racentreatinent interaction appryached significance (P less ^^an -.089) at the 
conventional .05 level on the reading Comprehension test. This was primarily 
bS> black-^experimental pupils tended to fare relatively better than white 

' • ' *Crainer,\liot and Sherin,'Ri chard J. MANOVA, Mujtivatiate Analysis of Vari;";f- . 

A program distributed by Clyde Computing Sec-^ice, Box 166, Coconut «rove Manon, 
,9^- Miami; Florida. .7. iO . 



experimental pupils. Acceptance of this hypothesis Is primarily a Judgmental 
rather than a statistical decision. A table of adjusted .reading. comprehension • 
mean scores ,4^ provided below for the (Elucidation of these who feel that this 
model may bes1»f1t the facts. The mean 'scores are adjusted for pre-test- differ- 
ences (as though all four groups scored the same on the pre-tests). 

TABLE III 

Adjusted Means For Reading Comprehension Interaction 
, Control Group #1 

Adjusted Mean (Read Cbinp) 



Race 


Treatment 


N 


White 


Exp. 


34 


White 


Cont. #1 


19 


Black 


Exp. 


48 


Black 


Cont if^ 


22 




One can see from 


the above 



24.3 
24.3 
24.7 
19.0 



The test of the simple treatment difference was signfficant (P less 
than .013) for the reading vocabulary test and approached significance for the 
comprehension test (P less than .067). ^A table of adjusted means is provided 
• below. 

.1 TABLE IV 

Adjusted Means for Tests of Simple Treatment Differences 
r * Conti*ol Group #1 

Group . JL Adjusted Mean (Read Voc) Adjusted Mean (Read Comp) , 

, Exp 82 ■ 23.1 24.. 7 

Cont #1 41 20.1 21.5-^ 

One can see- from the above that acceptance of these models leads to the 
conclusion that everyone, regardless of race or pre-test scores, tended to score 
higher if a member of "thie experimental rather than control group. This simple 
conclusion was clearly warranted by the sequence of statistical tests in th^e case 
of reading vocabularly scores. In the case of the comprehension test the only 
cpnclusion that was warranted, if one adheres strictly to the ,05 convention of 
statistical 'Significance, wis that the groups were not different. This would imply 
that parti eipationv. in the reading. center program was not associated with signifi- 
V cant differences in reading comprehension test results.' As was noted, it is real 
not a statistical matter as to Whether or not the models depicted in either of 
the tables, or the model involving no effect. Was the model which best fitted the 
facts concerning reading comprehension scores. > - 

EXPERIMENTAL GROUP VS.- CONTROL GROUP. #2 ' ( 

The hypothesis of equal regression coefficients could not be accepted 
in the case of the reading vocabulary test (P less than .042). This indicated 
that pupils' performance not only depended upon *their race and treatment group, 
but also upon how they scored .on the two pre-tests. Sudicomplax relatio^hips 
are difficult to clearly depict. Table V reflects a ra-the>\novel attempt to 
clarifys^the interaction. The first row in the table il lustrStes what the mean . 
scores on the vocabularly test would hav^^en had all groujjs scored the -same 
average on the two prertests as did the white ,exD|rimental. group (i.e. 23.3 and 
12.1). The secomi row shows the estimated' means^ terms of all groups receiving 



>. r-8- il 



the prc-tests as '*^lid the white control ^up (i.e. 



the same average scores on 
22.1 and 11 .5) and so on. 

The "If" columns depict; each of the four groups and report their pre-test 
scores. The "Then" columns show how each group would have fared under the circum- 
stances of having the same average pre-test \scores as each of the If groups. 
As can be noted from Table II. the bracketed diagonal entries are the average 
scores the groups actually received. 

TABLE V 

Estimated Vocabulary Scores based upon the Pre-Test Scores 
Received by each of the Race-Treatment Groups 
Control Group K2 

"THEN" 



^ ■ ■ "IP" ' Estimated Scores 

Group Pre Voc. F 're Comp . White Exp White Cont Black Exp Black Cqnt 

White Exp 23.3 12.1 . (22.7) ^2 2 . 22 9 21 .6 

White CoKt 22.1 11.5 • 2j.8 (22.1) , 22. 20.7 

Black Exp 21.0 12.9 ,, 22.6 20.9 (22.1) 20.0 

Black Cont 16.8 10.0 19.(5 17.5 19.0 (.6.8) 

The table indicates that the estimates for the black experimental pupils 
were, higher than for the black control pupils, given all four ^^tual permutations 
of the average pre-test scores. The differences were most pronounced under the 
circumstances of low scores on' both tests, (the black'-control group situation . 
Extension of these results beyond the.table would h^ve shown control group black* 
would- hav« been estimated to do. bettet than experimental group blacks only when 
their Ijre-test vocabulary scores were\+gh and pre-test comprehension scores low. 
If any black children actually existed who scored quite a bit higher on their 
initial reading vocabulary test than on their reading comprehension test, the 
above results would lead one to conclfide their progress in reading vocabularly 
would not have been helped by the reading center program. More generally, the 
benefits of the reading center in ternis of reading vocabularly seemed most pro- , 
nounced for black pupils with low scores on both pre-tests. 

Happily for the sake of clarity much the same sort of relationship . 
existed among the two white groups, but in a more pronounced way. In ^J^s pase 
also.white pupils with high pre-test\ocabularl v scores but low pre-test compre- 
hensionsco?es would apparently have been bettef off outside the ceriter. Conversely 
white pupils with low pre-test vocabulary scores, but high comprehension scores 
would have apparently benefited greatly from the program. . 

Table V .was constructed about actual permutations, of average scored, 
because orojecting to the above extremes would be meaningless since few or no - 
pupils a?Jually scored so differently w the two tests. The general conclusion 
from Table V would be that experimental group, whites who -scored low or> the^pr^- 
tests appeared to do relatively better after treatment than w;as the case with 
experimental groyp whites who scored high. on the pre-tests. : 

The reader must be warned that at this level of complexity, it may weHv 
be advisable to simply regard the above results as "noise". The improbable can v 
and will happen simpl]! as'a matter of chance If the P^^^"^?^^^^ f ::^,!:5P^^^^' ^ 
several* times, and the same complex interaction emerged eacMime,' it wou d 
wSl^rlnt serious consideration. Then one could use the results, for example, to 



^ J 



select pupiTs for project entrance who would benefit most from the program. 
Similarly, one wduld exclude from the project pupils who would be better off 
elsewhere.' As ma-tters stand,. the same pel ati.onship did not exist in comparisons 
with control group- #1. or comparisons involving the comprehension test. 'Jere is. 
no way to kn6w if' the relationship would emerge again in a repetition of the 
^dy. It jTiay be best "to interpret the results on. the reading vocabulary test 
in'Table V as: simplyyielding favorable projections for experimental group pupils, 
^6sp€x:iaily bTSicksrunder most realistic instances. '- . 

" The statistical tests involving reading comprehension scores.yi elded 

slightly more- clear-cut^ results. The interaction between race and treatment 
"approached significance (P less than .077). Table VI is provided as a service 
to readers who may vvilsh to further interpret this result. Since, a similar 
inteVaction approached significance in comparisons, involving control group #l, 
.more credence inay Be added to this possible interpretation. > 



^ . .. ^ TABLE VI 

Adjusted Means for Reading Comprehension Interaction 
■ ' Control Group #2. « 

/*. 



R|ce Treatment " " Adjusted -Means (Read Comp.)* : 

White : • Exp \: 34 22.7 - ^ . 

White Gortt if2 7 54 . 22.3 

qiack , Exp 48 -23.4 

Black ■ - Cont #2 41 18.6 

One can s$e from the above that acceptance of the interaction model 
„imjjfies that black- pupils seemed to have benefited more from the reading center, 
in the l.ong run than white pupils. 

The simple test of^main effect differences was significant (P less than 
.037)v. From a strictly statistical' standpoint, therefore, this model warrants^ 
more credehc^-*h«n the interaction model Table VII depicts adjusted means .f&r 
this model. ' . . 

TABLE VI.I 

J Adjusted Means For Simple Treatment Differences . 

Control Group #2 

Groups • . N Adjusted Means (Read Comp. ) 

Exp. V^. 23.2 ^. 

Cont #2 95 . ' ' 20.6 

On the above interpretation given equal pre-test scores, pupils of 
both#8ces scored higher, if in the experimental group. 



*The adjusted means for this table depended upon Jl^tingMJ^i^^on Jo/^^^^ren^^ 
datarS that used in constructing Table III. This ^sAgMeydj us ted means for 
the LxDerimental groups are not the same in the tv/o tabH^^S^ mam value of tne 
tablek'^??er?S shewing the size of contrasts, not in theWt values derived from 
the esVimalfing procedures. .The same applies to Tables IV- and VII. 

-10- : 




SUMMARY , ' - 

' •• ■ ■ - ' 

The basic trends in the data were favorable to project pupils. -, 

-Apparently, project benefits were not "washed out," even after the passage • 
of about a ye^r and one-half. Results were not sufficiently clear cut to 
warrant concluding that pupils "across the bmcd " benefited equally from the . 
program. There was some indication that lonPfcrm- results of participation 
might have been greater for blacks than whites, especially in the area of 
•reading comprehension. The. whUe-black interaction tendency was not as con-^ 
sistent in the area of reading vocabulary. One of the comparisons (contro] 
group #1) involving this area was favorable to reader center pupils regardless 
of race. However, differen^es^n pre-test scores as well as race,, which were, 
found to be associatltf-With reading center participation, -complicated the re- 
sults of the second comparisons. Whether such a complex relationship actually 
existed or was just due to "chance" conjunctions of data cannot be known. ;^In . 
any event, the overall trends in even the complicat;ed case were mostly favoraoie 
,t6 reading center |>artici pants . 

* 

The available evidence was basically sufficient to conclude that 
participation in the reading center program was probably associated with better 
long term test performance. The^complexities which appeared in the data, 
. especially those involving differential benefits for blacks, were not firmly 
enough established, by this study to warrant decision-making ^onc^. 



• APPENDIX B 
STATISTICAL TESTS 



\^ . ; / TABLE I . 
tests of Equality of Regression Coefficients: 

,/ First Control vs Experimental 

TEST OF / T P less than 
Read Voc . ' . - 

Read Voc. / 6,111 1.18 .322 

Read Comp.. . , 6,111 . • .80 * - .573 

Second Control vs Experimental 

Read Voc. / 6,165 2.24 .042* 

Read Comp.- ,6,165 .90 .493 

TABLE II . 
Tests of Treatment - Race Interaction: 



Fir^t 



Control vs. Experimental 
TEST . DF . ' P less than 



r 



Read Vot. 1,11-7 . .93 

Read Comp. 1 ,117 , 2.94 

■; , . ' f 

^ , Second Control vs. Experimental 

.Read"Voc. ' 1,171 1.57 • .212 

Read Comp .1,117 3.17 / .077 

. . TABLE III ° 

Tests of Treatment Main. Effect: 

, First "Control vs. Experimental 

TEST . . DF £ ' P less than 

Read Voc: . I,'ll7 6.40, . .013 

Read Comp. 1,117 , 3.41 ' .067 » 

. ■ Q . Second Control vs. Experimental ■ 

Read Voc. 1.171 .80 , ' .373 | 

Read Comp. 1,171 4.41 .037 

♦Since the interaction was significant at' the .Q5 level, the remaining analyses 
of reading vpdabulary scores were superfluous fronv-the standpoint of the MANOVA 
logic. In brief, the hypothesis that the interaction model held could not be 
rejected. ^ ^ I 



